Use of eye tracking for tool identification and assignment in a robotic surgical system

ABSTRACT

A robotic surgical system includes an eye gaze sensing system in conjunction with a visual display of a camera image from a surgical work site. Detected gaze of a surgeon towards the display is used as input to the system. This input may be used by the system to assign an instrument to a control input device (when the user is prompted to look at the instrument), or it may be used as input to a computer vision algorithm to aid in object differentiation and seeding information, facilitating identification/differentiation of instruments, anatomical features or regions.

TECHNICAL FIELD OF THE INVENTION

The present invention relates generally to the use of eye tracking insurgical robotic systems.

BACKGROUND

US Published Application No. 2013/0030571 (the '571 application), whichis owned by the owner of the present application and which isincorporated herein by reference, describes a robotic surgical systemthat includes an eye tracking system. The eye tracking system detectsthe direction of the surgeon's gaze and enters commands to the surgicalsystem based on the detected direction of the gaze.

FIG. 1 is a schematic view of the prior art robotic surgery system 10 ofthe '571. The system 10 comprises at least one robotic arm which actsunder the control of a control console 12 managed by the surgeon who maybe seated at the console. The system shown in FIG. 1 includes multiplerobotic arms 11 a, 11 b, 11 c. Three such arms are shown but a larger orsmaller number may be used. Each robotic arm can support and operate asurgical instrument 14, 15, 16 for use on a patient 13. One of theinstruments 14 is preferably a camera which records the operating fieldinside the patient, while the other instruments may be known surgicaltools 15, 16.

The arms 11 a, 11 b, 11 c are operated by an electronic control unit 30which causes the arms to perform the movements entered via the console12. The unit 30 will receive the high-level movement commands (forexample, desired position and inclination of the tool supported by therobot) and will execute them, converting them into the correspondingsequences of signals to be sent to the individual motors of the robotarm articulations. Other details of the system 10 are found in the '571application which is fully incorporated herein by reference.

The console includes input devices 17, 18 which can be gripped by thesurgeon and moved so as to deliver instructions to the system as to thedesired movement and operation of the instruments supported by the arms11 a, 11 b, 11 c.

The surgeon's movements are suitably reproduced by the surgicalinstruments by means of movement of the robotic arms. The input devicesmay be equipped to provide the surgeon with tactile feedback so that thesurgeon can feel on the input devices 17, 18 the forces exerted by theinstruments on the patient's tissues.

Each input device will typically operate a robot arm. The '571application describes that where there are two input handles and morethan two arms carrying instruments, the system includes a control on theconsole that allows the surgeon to assign each arm to a desiredinstrument. This allows a surgeon to control of two of the surgicalinstruments disposed at the working site at any given time. To control athird instrument disposed at the working site, one of the two handles17, 18 is operatively disengaged from one of the initial two instrumentsand then operatively paired with the third instrument.

The console may also include a keyboard 19 and/or touch screen and/orother command input devices. These other command devices might include apedal device 20, and a button(s) on or in proximity to one or bothhandles of the input devices 17, 18.

The console 12 has an eye movement tracking system 21 or so-called “eyetracker” for detecting the direction of the surgeon's gaze towards theconsole and for controlling the surgical system depending on the gazedirections detected. In this way, the surgeon may control functions ofthe system by means of movement of his/her eyes.

The eye tracking system estimates the direction of the surgeon's gazetowards the display 22 and performs selection of the commands associatedwith a zone when it detects a gaze direction which falls within thiszone. In one particular example described in the '571, the commandsassociated with selection areas 29 on the display 22 comprise thecommands for assigning particular ones of the arms 11 a, 11 b, 11 c tothe surgeon input devices 17, 18. That allows the surgeon to alternatecontrol of the robot arms on the two input devices without letting go ofthe input devices, but instead by simply looking at the correspondingselection areas on the screen. For example, while controlling each ofthe arms 11 a, 11 c with one of the input devices 17, 18, the user mightre-assign input device 17 over to arm 11 b in order to use or repositionthe instrument 9 b within the body. Once the task involving movement ofinstrument 9 b is completed, the surgeon can rapidly re-assign inputdevice 17 back to robot arm 11 a. These steps can be performed by usingthe eye tracking features to “drag and drop” icons on the consoledisplay towards icons representing the various arms.

In another example described in the '571, the eye tracking system isused to move the camera based on where the surgeon is looking on thedisplay 22. When this function is enabled (e.g. by entering an inputcommand, such as through pressing of a button on the console, depressinga foot pedal, etc.), the movement of the eyes over the image of theoperating field on the screen causes the movement of the robot armsupporting the camera. This can be used to place the zone the surgeon isfocused on at the center of the display screen.

The '571 also describes use of the eye tracker to detect the distancebetween the screen and surgeon's eyes as a way to allow the surgeon to“zoom” the camera display in or out. The system enlarges the picture ofthe operating field shown on the screen depending on a variation in thedistance detected. With this feature, the surgeon can intuitivelyperform enlargement of the picture by simply moving his/her face towardsthe screen and, vice versa, increase the viewing area of the operatingfield, thus reducing enlargement, by moving his/her face away from thescreen.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of a robotic surgery system.

FIG. 2 depicts an image on an endoscopic display, and an overlayindicating the user's gaze location with respect to the display;

FIG. 3 is similar to FIG. 2, but additionally shows a prompting overlay;

FIG. 4 is similar to FIG. 3, but the prompting overlay has disappeared,and an additional overlay is shown over the shaft of one of theinstruments;

FIG. 5 depicts an image on an endoscopic display, together with a firstcolor overlay G on the left side of the screen and a second coloroverlay B on the right side of the screen;

FIGS. 6-8 are a sequence of drawings depicting an image on an imagedisplay together with overlays reflecting use of a contour model toidentify boundaries of an instrument;

FIGS. 9-10 are a sequence of drawings depicting an image on an imagedisplay together with overlays reflecting use of a region growingalgorithm to identify regions of an instrument;

FIG. 11 depicts an image on an endoscopic display, in which a brightregion marks a position of a ureter having an illumination device withinin it, and in which a first overlay marks a first seed location placedusing eye tracking input and used in identifying contours of theilluminated ureter;

FIG. 12 is similar to FIG. 12 and shows a contour that follows thelighted ureter after image segmentation.

FIG. 13 is a flow diagram illustrating a first method of using eyetracking input to aid in assignment of a user input devices to andinstrument or its corresponding robotic arm.

FIG. 14 is a flow diagram illustrating a second method of using eyetracking input to aid in assignment of a user input devices to andinstrument or its corresponding robotic arm.

FIG. 15 is a flow diagram illustrating a method of using eye trackinginput to aid in computer recognition of an object displayed on anendoscopic image display.

DETAILED DESCRIPTION

This application describes a system having features allowingdifferentiation of objects or regions on a displayed image, such as asurgical site, using eye gaze sensing as an input. In particularembodiments, it allows the use of eye tracking within a displayed imageto aid in assignment of instruments to robotic manipulators, and/or toaid the system in using computer vision to recognize instruments shownon the endoscopic display.

System

The system includes elements described in the Background section andshown in FIG. 1, namely at least one robotic manipulator or arm 11 a, 11b, 11 c, at least one instrument 15, 16 positionable in a work spacewithin a body cavity by the robotic manipulator or arm, a camera 14positioned to capture an image of the work space, and a display 23 fordisplaying the captured image. An input device 17, 18 or user controlleris provided to allow the user to interact with the system to give inputthat is used to control movement of the robotic arms and, whereapplicable, actuation of the surgical instrument. An eye tracker 21 ispositioned to detect the direction of the surgeon's gaze towards thedisplay.

A control unit 30 provided with the system includes a processor able toexecute programs or machine executable instructions stored in acomputer-readable storage medium (which will be referred to herein as“memory”). Note that components referred to in the singular herein,including “memory,” “processor,” “control unit” etc. should beinterpreted to mean “one or more” of such components. The control unit,among other things, generates movement commands for operating therobotic arms based on surgeon input received from the input devices 17,18, 21 corresponding to the desired movement of the surgical instruments14, 15, 16.

The memory includes computer readable instructions that are executed bythe processor to perform the methods described herein. These includemethods of using eye tracking input in a sequence for assigning userinput devices to selected surgical instruments or robotic manipulators,and methods of using eye tracking input in a sequence for recognizingsurgical instruments positioned in a surgical work site and displayed onan endoscopic display.

Assigning User Inputs to Instruments/Robotic Manipulators

An exemplary system includes a mode of operation that allows the user tolook at an instrument displayed in an endoscopic image to initiateassignment of that instrument to a given hand controller at the surgeonconsole. FIG. 13 is a flow diagram showing a method that can beperformed by the processor during this mode of operation, in accordancewith the computer readable instructions, to assign a hand controller toan instrument based on user eye gaze input. FIGS. 2-4 depict an imagedisplay displaying an endoscopic image 100 of a work area with in a bodycavity during the performance of this method. The displayed imageincludes two instruments 101 and the surrounding tissue within the bodycavity.

In general, the method starts with the system entering into aninstrument pairing sequence. This can be initiated by the user or bepart of an initial set-up sequence performed at the start of aprocedure. As depicted in FIG. 13, the steps performed by the processorinclude receiving eye gaze information from the eye tracker. Step 200.This input allows the processor to determine the region of the screenthe user is viewing, and thus allows it to determine, in Step 202 whichof the instruments 101 on the display the user is looking at. As shownin FIGS. 2 and 3, a visual overlay 102 may be caused by the processor tobe generated and displayed to provide feedback to the user about thecurrent gaze location. The overlay is depicted as a dashed circle in thefigures but may be generated in any form useful to the user. Whendisplayed, the overlay will track the user's gaze on the endoscopicimage.

In an optional step depicted in FIG. 3, the user is cued using visual orother prompts 14 to look at the instrument within the view to assign itto a given input controller. A visual prompt might be an overlay causedby the processor to be generated and displayed on the image display asshown. In the example shown in FIG. 3, the overlay cues the user with aninstruction to look at the instrument to be paired with the user inputdevice that is to be operated by the user's left hand. Other promptsmight include auditory prompts, or tactile prompts such as vibration ofthe hand controller that is to be paired in the pairing sequence. In thesequence of steps depicted in FIG. 13, this step is performed prior tothe step of determining which instrument on the display the user islooking at.

In an alternative embodiment, rather than being prompted, the user mightinstead input instructions to the processor directing the processor toenter a sequence of pairing a designated one of the hand controllerswith an instrument. For example, if the user wishes to pair the handcontroller on the user's right, the user might, after instructing thesystem to enter an instrument pairing sequence, first give input to thesystem that it is the right-hand controller that is to be paired. Thismay be done using an input feature on the hand controller itself, suchas a button, knob or other switch. Once this input has been given to thesystem, the method proceeds with Step 200.

The step of determining which instrument the user is looking at may beperformed using various methods. For example, when the user looks at theinstrument to be assigned to a particular controller, the system mayemploy a computer vision algorithm to differentiate that instrument fromthe surrounding features in the camera image. Some computer visionalgorithms that may be used for this purpose are described below, butothers can be used as alternatives. In some embodiments, the instrumentposition(s) may be known in the endoscopic view using other means. Forexample, based on kinematic information the processor receivesinformation defining, or can determine, the relative positions andorientations of the camera to the instruments within the body cavity.This allows, in Step 202, a determination of which instrument on thecamera view displayed is the one the user is looking at. The system mayapply a visual overlay 106 to that instrument, such as the one displayedover the image of the instrument shaft in FIG. 4, to confirm to the userthat the instrument has been identified.

After it has been determined which instrument the user is looking at,that instrument is assigned to the user input device/hand controller.Step 204. If there are multiple user input devices, the user will havebeen prompted, as discussed in connection with FIG. 3, or will haveinstructed the system, as to which user input device is in the processof being assigned. Once the eye-selected instrument is paired with therelevant user input device, the processor can, upon receiving instrumentinput from that user input device (Step 206), transmit robot controlsignals that cause the robotic arm supporting the eye-selectedinstrument to manipulate and/or actuate that instrument.

The described sequence may be a one that is performed before system use,and it may also be performed at a later time during the surgicalprocedure. For example, it can be used to re-assign instruments orchange pairings of instruments and user input devices mid-procedure, orto assign instruments based on repositioning of robotic arms or thepatient. In some procedures the system includes a fourth manipulator armhandling a third instrument. In such procedures this assignment may alsobe used to swap control of an input controller from a one instrument toanother instrument that is not presently assigned to an inputcontroller.

A second embodiment is one in which the instrument position(s) in theendoscopic view is/are already known or may be acquired. In accordancewith this method, the system is configured such that detecting theuser's gaze to a particular area of the screen (e.g. the left or rightside) may be sufficient input to instruct the system to assign theinstrument in that area to the appropriate hand/input controller. Thismethod is depicted in the flow diagram of FIG. 14 and illustrated inFIG. 5. Although FIG. 5 uses the left and right halves of the screen asthe defined area, smaller regions of the display may also be used.Overlays may be used to provide feedback to the user confirming theassignment, such as by briefly showing overlays above or on the assignedinstrument or its region on the screen. For example, overlays 108 maycover the left and right sides of the screen, with each overlay being ina differentiating color (shown in FIG. 5 is green (G) on the left andblue (B) on the right) of the robotic manipulator arm holding theassigned instrument. In this figure, the overlaid colors inform the userthat the “green arm” is holding the left-hand instrument, and the “bluearm” is holding the right-hand instrument. This overlay may be displayedtemporarily or may change saturation or form/size to be less visuallyobstructing to the surgical scene.

FIG. 14 is a flow diagram showing this method, which can be performed bythe processor according to the computer readable instructions. Themethod begins with the system entering into an instrument pairingsequence. This can be initiated by the user or be part of an initialset-up sequence performed at the start of a procedure. Eye gazeinformation is received from the eye tracker. Step 300. This inputallows the processor to determine the region of the screen the user isviewing, Step 302. Because the processor has, or can obtain ordetermine, information as to which area of the display one or bothinstruments are in, it can determine, in Step 304 which of theinstruments 101 on the display is in the area the user is looking at. Asdiscussed in connection with FIG. 5, visual overlay may be caused by theprocessor to be generated and displayed to provide feedback to the userabout the identified area of the display or the gaze location.

In an optional step, the user may be cued in a manner similar to thatdescribed with respect to FIG. 3, using visual or other prompts to lookat the instrument within the view to assign it to a given inputcontroller. A visual prompt might be an overlay caused by the processorto be generated and displayed on the image display as shown. In theexample shown in FIG. 3, the overlay cues the user with an instructionto look at the instrument to be paired with the user input device thatis to be operated by the user's left hand. Other prompts might includeauditory prompts, or tactile prompts such as vibration of the handcontroller that is to be paired in the pairing sequence. In the sequenceof steps depicted in FIG. 14, this step is performed prior to the stepof determining which area of the display the user is looking at.

In an alternative embodiment, rather than being prompted, the user mightinstead input instructions to the processor directing the processor toenter a sequence of pairing a designated one of the hand controllerswith an instrument. For example, if the user wishes to pair the handcontroller on the user's right, the user might, after instructing thesystem to enter an instrument pairing sequence, first give input to thesystem that it is the right-hand controller that is to be paired. Thismay be done using an input feature on the hand controller itself, suchas a button, knob or other switch. Once this input has been given to thesystem, the method proceeds with Step 300.

The step of determining which instrument is in the area the user islooking at may be performed using various methods. Although computervision algorithms described with respect to the first embodiment can beused, this embodiment is well suited to systems in which the instrumentposition(s) are known or can be determined in the endoscopic view usingother means. For example, based on kinematic information the processorreceives information defining, or from which it can determine, therelative positions and orientations of the camera to the instrumentswithin the body cavity. This allows, in Step 304, a determination ofwhich instrument on the camera view displayed is the one in the regionthe user is looking at. The system may apply a visual overlay to theregion the user is looking at, or on the identified instrument (based onits known position relative to the camera), or give some other feedbackto the user, to confirm to the user that the instrument has beenidentified.

After it has been determined which instrument the user wants to pairwith the relevant hand controller, that instrument is assigned to theuser input device/hand controller. Step 306. If there are multiple userinput devices, the user will have been prompted, as discussed inconnection with FIG. 3, or will have instructed the system, as to whichuser input device is in the process of being assigned. Once theeye-selected instrument is paired with the relevant user input device,the processor can, upon receiving instrument input from that user inputdevice (Step 308), transmit robot control signals that cause the roboticarm supporting the eye-selected instrument to manipulate and/or actuatethat instrument.

The described sequence may be a one that is performed before system use,and it may also be performed at a later time during the surgicalprocedure. For example, it can be used to re-assign instruments orchange pairings of instruments and user input devices mid-procedure, orto assign instruments based on repositioning of robotic arms or thepatient. In some procedures the system includes a fourth manipulator armhandling a third instrument. In such procedures this assignment may alsobe used to swap control of an input controller from a one instrument toanother instrument that is not presently assigned to an inputcontroller.

It should be understood that the use of eye tracking input to select aninstrument may be used for other purposes besides eye tracking. Forexample, the eye tracking input may be used to select an instrument sothat some particular action or function can be performed using thatinstrument, or so that that instrument can be placed in a predeterminedoperational mode. Actions or functions that could be performed include,without limitation, any of the following:

Clutching—the processor causes the selected instrument to be placed in aclutched state in which movement of the user input device with whichthat instrument is paired is temporarily suspended, allowing the user toreposition to user input device for ergonomic or other reasons;

Device Activation—the processor causes the selected instrument todeliver energy (electrical, ultrasonic, thermal, etc.) to the tissue, orto deliver staples, clips or other fasteners to the tissue, or to clampagainst the tissue;

Semi-Autonomous or Autonomous modes of operation—the processor causesthe selected instrument to enter into semi-autonomous modes or operationor otherwise perform autonomous or semi-autonomous actions. For example,the eye-selected instrument may be placed in a mirrored motion ormatched motion mode of the type described in co-pending U.S. Ser. No.16/236,636, filed Dec. 30, 2018 (“Dynamic Control of SurgicalInstruments in a Surgical Robotic System”), or caused to applycounter-traction as described in co-pending PCT/US2018/031916 (“Systemand Method for Modulating Tissue Retraction Force in a Surgical RoboticSystem”), or to return to a predetermined home position or togglebetween two predetermined positions for repetitive tasks.

Graphically tagging and recalling identified structures, as described inco-pending application Ser. No. 16/018,037 (“Method of GraphicallyTagging and Recalling Identified Structures Under Visualization forRobotic Surgery”).

Using Eye Tracking Input to Assist Computer Recognition of SurgicalInstruments

User gaze information may be used as input to a computer visionalgorithm to aid the steps of in differentiating/segmenting aninstrument or other object displayed on the endoscopic display from itsenvironment. Image segmentation is a method in which an image isseparated into regions corresponding to contours or objects of interest.In the disclosed system the gaze location may identify to the computervision algorithm the region of interest for the instrument recognitionalgorithm, or it may identify meaningful edges of the instrument to berecognized. The system then employs differentiation/segmentationalgorithms to detect boundaries of the instrument.

Several methods may be used for image segmentation. Some methods arebriefly described here merely by way of example. These will be describedwith respect to recognition of surgical instruments but can also be usedto identify other objects within the work site.

FIGS. 6-8 illustrate one example in which an active contour model(sometimes called a snake) is facilitated by eye gaze information. Aflow diagram depicting the method performed by the processor is shown inFIG. 15. To begin, the user may be prompted to look at a region of theinstrument that is to be recognized. The eye tracker detects the user'sgaze towards a particular instrument on the camera display. FIG. 15,Step 400. As described in connection with the earlier embodiments, anoverlay 102 may be generated to display the area on the screen that theuser us looking at.

The eye gaze input from the eye tracker is used to generate a “seed.”FIG. 15, Step 402. An overlay 110 may be displayed as in FIG. 7,representing the seed. This seed is recognized by the system as beingwithin the bounds of the instrument due to the fact that the user hasdesignated it to be as such using the eye tracking input. Imagesegmentation is next performed to recognize the boundaries of theinstrument. FIG. 15, Step 404. FIG. 7 shows an initial state of oneexample of image segmentation using the active contour model. A line Lwith a plurality of nodes is shown. The illustrated initial placement ofthe contour line is made based on the basic assumption that oneinstrument is on the left side of the image and the other instrument ison the right side of the image. The contour moves through the imagetowards the seed, detecting edges of the instrument and conforming tothose edges, thus creating a contoured region that contains the seedlocation. Parameters of “tension” between the nodes/contour shapeparameter will guide the motion of the contour as it moves through theimage and encounters edges. Note that the graphics shown in FIGS. 6-8 toillustrate use of the model may not be shown to the user, although theuser may be shown some simplified version.

In another embodiment, the program may use gaze information to generatemultiple seeds in a region-growing algorithm as an alternate, oradditional, method of image segmentation. For example, as shown in FIG.9, a seed location is identified using eye gaze as specified withrespect to the prior example. It may be marked by an overlay 110 asdiscussed. Then the system grows the region to surrounding parts of theinstrument as illustrated by the shaded region 112 around the seed inFIG. 9. Continued detection in each direction relative to the seedcontinues to expand the edge of the contour to the detected boundariesof the instrument in the image. FIG. 10 shows the detected regionexpanded closer to the boundaries of the instrument.

The shape of the region used for this model may be a simple geometricfigure such as the rectangular region shown, a more complex polygon, orsome other combination of lines/shapes/curves. An amorphous region thatexpands to the boundaries of the instrument may also be used; an exampleof this may be an active contour model whose initial shape lies withinthe instrument and expands to its boundaries. In others, more complexshapes or contours may be used. The complexity of the shape used maydepend on the intended task. Thus, using a simple geometrical shape asshown may be sufficient for an instrument assignment task, but othertasks requiring more complex shape determination and differentiation(e.g. use for collision detection or surgical task assurance describedin U.S. Ser. No. ______, filed Dec. 31, 2018 “System and Method forControlling a Robotic Surgical System Using Identified Structures”) mayrequire more complex shapes or contours.

Other image segmentation means are also within the scope of thisinvention, and combinations of image segmentation means may also be usedto improve performance.

Other uses of eye tracking may include the use of eye tracking input todifferentiate between other structures in the image instead of, or inaddition to, instruments. For example, in FIG. 11, a lighted ureteralstent is used to provide better visualization of the ureter. This isuseful because the ureter is typically hidden below fascia and yet is astructure the surgeon wishes to avoid contacting with instruments toavoid ureteral injuries while operating deep in the pelvis. In thisfigure, brighter region 114 marks the lighted ureter.

The user may be prompted to look at the object to be identified. Theprompt may be an overlay display an instruction (e.g. “look at thelighted ureter”) or some other form of prompt. The system then detectsthe users gaze (FIG. 15, Step 400) in order to provide seed informationto a computer vision algorithm. See FIG. 11, and FIG. 15, Step 402.Active contour models, region-growing algorithms, or other means asdiscussed above may be used for image segmentation. FIG. 15, Step 404.FIG. 12 schematically illustrates the initial seed 116 identified usingeye tracking input and a contour following the lighted ureter as aresult of image segmentation. The process may be repeated for otherinstruments within the field. For example, new seed location may becreated by tracking the user's gaze as the user looks at the image ofanother surgical instrument together, and contour thus being determinedto follow the shape of the instrument.

In modified versions of the computer recognition embodiments, the usermay give input to the system instructing the system to enter into anobject-recognition or instrument-recognition mode. Following receipt ofthat input, the system will receive the eye gaze input to begin theprocess of object recognition based on the eye gaze input.

Multiple cues for a single structure, object, instrument, or region maybe used to provide input to the computer vision algorithm. A pluralityof prompts such as “Look along the length of the instrument”, or “Lookat the tip of the instrument” and “Look at the proximal end of theinstrument shaft” may be used to initiate collection of eye gaze inputfrom which multiple seed locations can be generated, or to initiatecollection of eye gaze input that indicates a general direction ofmotion that cues the computer vision algorithm and assists withsegmentation. The system may give a sequence of prompts in order to gaina plurality of eye gaze inputs to the instrument marking different partsof the instrument. An overlay may provide a visual indication of thelocation of the user's gaze as discussed with respect to FIG. 2.

The system may use a 2D or 3D camera. Use of a 3D camera may provideadditional stereoscopic information that, when seeded with the eyetracking input, provides even more robust results from a computer visionalgorithm.

The eye tracking input may be used to define planes of interest, regionsof interest, structures or regions to avoid, or structures/planes tofollow for use in methods of the type described in U.S. Ser. No. ______,filed Dec. 31, 2018 “System and Method for Controlling a RoboticSurgical System Using Identified Structures”, and U.S. Ser. No.16/010,388, filed Jun. 15, 2018 (“Method and Apparatus for Trocar-BasedStructured Light Applications”), or it may be paired with otherinformation, such as kinematic models, to assist a computer visionalgorithm as also described in that application.

All patents and applications referred to herein, including for purposesof priority, are incorporated herein by reference.

I claim:
 1. A robotic surgical system, comprising: at least two surgicalinstruments, each instrument moveable by a robotic manipulator within awork area; a camera positioned to capture an image of a portion of thework area; an image display for displaying the image; a user inputdevice; an eye gaze sensor positionable to detect a direction of theuser's gaze towards the image of the work area on the display, and aprocessor configured to determine, based on the direction detected bythe eye gaze sensor, which of the at least two instruments the userwants to assign to the user input device, and to assign the user inputdevice to the determined instrument.
 2. The system of claim 1, whereinthe processor is configured to receive input from the user input deviceand to control the robotic manipulator to move the instrument inresponse to the input.
 3. The system according to claim 1, wherein theprocessor is configured to determine, based on the direction detected bythe eye gaze sensor, which of the at least two instruments the user isviewing on the image display, and to assign the user input device to aninstrument identified to the system using detected eye gaze.
 4. Thesystem according to claim 1, wherein the processor is configured todetermine, based on the direction detected by the eye gaze sensor, theregion of the image display of the work site that is occupied by theinstrument the user wants to assign to the user input device.
 5. Arobotic surgical system, comprising: a surgical instrument moveable by arobotic manipulator within a work area; a camera positioned to capturean image of a portion of the work area; an image display for displayingthe image; a user input device; an eye gaze sensor positionable todetect a direction of the user's gaze towards the image of the work areaon the display, and a processor configured to execute a computer visionalgorithm to detect at least a portion of the instrument displayed inthe image using input from the eye gaze sensor.
 6. A method of using arobotic surgical system, comprising: positioning at least two surgicalinstruments, within a work area, each instrument moveable by a roboticmanipulator; capturing an image of the surgical instruments within thework area; displaying the image on an image display; receiving from aneye gaze sensor input representing a direction of a user's gaze towardsthe image, and determining, using the eye gaze sensor input, which ofthe at least two instruments the user wants to assign to a user inputdevice, and assigning the user input device to the determined instrumentto enable control, using input from the user input device, of therobotic manipulator operatively associated with the determinedinstrument.
 7. The method of claim 6, wherein the method furtherincludes the step of receiving input from the user input device andcontrolling the robotic manipulator to move the determined instrument inresponse to the input.
 8. The method according to claim 6, wherein thedetermining step includes determining, based on the direction detectedby the eye gaze sensor, which of the at least two instruments the useris viewing on the image display, and wherein the assigning step includesassigning the user input device to an instrument identified to thesystem using detected eye gaze.
 9. The method according to claim 6,wherein the determining step includes determining, based on thedirection detected by the eye gaze sensor, the region of the imagedisplay of the work site that is occupied by the instrument the userwants to assign to the user input device.
 10. A method of using arobotic surgical system, comprising: positioning a surgical instrumentmoveable by a robotic manipulator within a work area; capture an imageof a portion of the work area; displaying the image on an image display;receiving from an eye gaze sensor input corresponding to a direction ofa user's gaze towards the image of the work area on the display, andexecuting a computer vision algorithm to detect at least a portion ofthe instrument displayed in the image using input from the eye gazesensor.
 11. A method of detecting an object within a surgical work site,comprising: capture an image of a portion of a surgical work area withina body cavity; displaying the image on an image display; receiving froman eye gaze sensor input corresponding to a direction of a user's gazetowards the image of the work area on the display, and executing acomputer vision algorithm to detect at least a portion of an objectdisplayed in the image using input from the eye gaze sensor.
 12. Themethod of claim 11, further including the step of, prior to thereceiving step, prompting the user to look at the object on the imagedisplay.
 13. The method of claim 11, further including the step of,prior to the receiving step, receiving a user instruction to recognizean object visible on the image display.